Robustly representing uncertainty through sampling in deep neural networks

As deep neural networks (DNNs) are applied to increasingly challenging problems, they will need to be able to represent their own uncertainty. Modelling uncertainty is one of the key features of Bayesian methods. Using Bernoulli dropout with sampling at prediction time has recently been proposed as an efficient and well performing variational inference method for DNNs. However, sampling from other multiplicative noise based variational distributions has not been investigated in depth. We evaluated Bayesian DNNs trained with Bernoulli or Gaussian multiplicative masking of either the units ar X iv :1 61 1. 01 63 9v 4 [ cs .L G ] 1 S ep 2 01 7 (dropout) or the weights (dropconnect). We tested the calibration of the probabilistic predictions of Bayesian fully connected and convolutional DNNs on two visual inference tasks (MNIST and CIFAR-10). Sampling at prediction time increased the quality of the DNNs’ uncertainty estimates. Sampling weights, whether Gaussian or Bernoulli, led to more accurate representation of uncertainty compared to sampling of units. However, sampling units using either Gaussian or Bernoulli dropout led to increased convolutional neural network (CNN) classification accuracy. Based on these findings we used both Bernoulli dropout and Gaussian dropconnect concurrently, which we show approximates the use of a spike-and-slab variational distribution without increasing the number of learned parameters. We found that spike-and-slab sampling efficiently combined the advantages of the other methods: it classifies with high accuracy and robustly represents inferential uncertainty for all tested architectures.